How to BackUp and Shrink Your Raspberry Pi Image

Foreword

So why all the hullabaloo you may ask?

Surely all I need to do is use Win32DiskImager and take a read of my Raspbian Image and zip it right?

Job Done!

Well, unfortunately that's not the whole story. Read on...

Have you ever noticed the size of your Rasbian image is that of the card you took the copy from and does not reflect the size actually being used?
- This happens when you re-size your image (via raspi-config) to make use of the full uSD card even if you end up not using much of the card space, the saved image will still be the same size as the uSD card.
Have you ever found Win32DiskImager fails to write a backup Raspbian image even when you are writing to a 'like for like' uSD card of the same stated size, just a different make?
- This is because not all card sizes are the same ie. a 16GB uSD card from one manufacturer may not be exactly the same size as from another. This will cause Win32DiskImager to fail.

Did you want to downsize your Raspbian image from one card to another but can't?
- If you use the Win32DiskImager read method you can only write to the exact same or bigger card size.

The following text details how you can make more efficient backups of your images.

Introduction

As mentioned above this Instructable documents how to efficiently make backups of your critical Raspberry Pi images, compress them such that they occupy the least possible space and can potentially be transferred to a uSD card of smaller size.

The Instructable will cover the following topics;

How to install prerequisite software on Raspberry Pi to allow image resizing.
How to read and store an Image from a uSD card (via Linux command line and with Win32DiskImager),
How to create your compressed backup image,
How to write your backup image to a uSD card (via Linux command line only. it assumes you know how to use Win32DiskImager to write an image. If not go here https://www.raspberrypi.org/documentation/installa...),
Summary,
References used.

What parts do I need?

To set the system up such that you can successfully carry out the instructions you will need the following;

Raspberry Pi 3 + Jessie Image installed (in this case I used : 2016-03-18-raspbian-jessie.img) with monitor, mouse and keyboard etc.,
uSD USB card reader for your uSD card to be backed up,
External USB HDD for Raspberry Pi (big enough to hold your backed up images, in this instance mine is labeled '1TBFREECOM' and oddly enough is 1TB in size, which is more than ample for the needs of this Instructable),
Target uSD card to read the uncompressed Image for backup,
Target uSD card to write the compressed Image to (could be the same as 4.).

What skills do I need?

Do I require any specialist skills?

No, only the following;

Some knowledge of Linux is always useful,
A little patience as the writing process may take a while,
The ability to accurately and blindly follow written instruction.

Note

This method is only good for Raspbian images having two partitions, so won't work with NOOBS, RetroPi etc.
It is assumed that all Linux commands are carried out directly on the Raspberry Pi (not via a PuTTY SSH connection etc.).
Linux commands are entered via a Terminal instance. Terminal is the 'black monitor' icon on the start bar. Alternatively, it can be found via; Menu -> Accessories -> Terminal.
Although uSD (micro SD) is used throughout, this method applies equally to SD cards.
This was written to complement my series on home automation https://www.instructables.com/id/IoT-Intranet-of-Th...

Disclaimer

As always, you use these instructions at your own risk.

Step 1: Initial Raspberry Pi Software Install

Before we can begin we must install the following items of software. This is a 'one off' activity and won't take long.

Enter the following Linux commands, answering 'y' to any questions;

sudo apt-get update
sudo apt-get install dcfldd
sudo apt-get install gparted

Once complete you have now installed the software necessary to write an image and change the partition of an image.

Step 2: Reading Your Raspberry Pi Image

Via Win32DiskImager

As in picture one above;

Insert your uSD card into your PC USB reader and note which drive it is mapped to. In the example above it is drive 'G:\',
Enter the directory where you want the Image to be written and give it a name. Here : 'c:\temp\RaspberryPi\BackUpServerUncompressed.img',
Click the read button. Circled in red,
Let the software complete,
The Image file located in 'c:\temp\RaspberryPi\' can now be moved to the 1TB FREECOM External HDD for transfer to the Raspberry Pi.

That's it you're done.

or Via the Linux command line

This is a little bit more involved.

Insert the Raspberry Pi Image to be copied into your USB uSD card reader of your Raspberry Pi.

Insert your USB External HDD into your Raspberry Pi.

Open a Terminal instance and enter the following Linux command;

df -h

This is to determine how your uSD card has been mounted (connected/mapped).

See Pic 2 above for the result. In general you will find external devices have a path something like '/media/pi/...'

In this case;

For the External HDD (named '1TBFREECOM') it is '/media/pi/1TBFREECOM'
For the Raspberry Pi Image there will be two entries under '/media/pi/...'
- One named 'boot'
- The second will have a long number, here 'e6e7f776-...-c44ecdbfcf90'

For these two Raspberry Pi entries note the mount details. Ringed in red, here; '/dev/sdb1' and '/dev/sdb2'

Next we 'unmount', the Raspberry Pi uSD card with the following Linux command (see pic 3).

sudo umount /dev/sdb1 /dev/sdb2

Now we make a backup copy of the Raspberry Pi image to our external HDD, named '1TBFREECOM'

sudo dcfldd if=/dev/sdb of=/media/pi/1TBFREECOM/MyImage.img

Pic 4 shows the backup activity in action and Pic 5 shows what it looks like when it's complete.

Next we use the sync command to force a synchronise of any outstanding input or output (see pic 6).

sudo sync

Finally we finish off by taking a look at the backed up image file on the 1TBFREECOM HDD to check to see that user name and group are 'pi'.(see pic 7 above. User and Group are circled in red).

ls -l /media/pi/1TBFREECOM

If not, then enter the following command (see pic 8);

sudo chown pi:pi /media/pi/1TBFREECOM/MyImage.img

And that's all there is to it!

Ok, so I admit it's a little on the tortuous side.

.
.

Gotchas

In case your image is sizable, you have to format your USB-FDD/-stick as NTFS (example: image of 16 GB on a 32 GB stick). Linux/Jessie distros have a NTFS-driver, which is read-only. Therefore you need to install the newer driver NTFS-3G, which is read & write capable. sudo apt-get install ntfs-3g and reboot.

Thanks to gorgonops for spotting this. See in line comments below for thread.

Step 3: Resizing Your Image

Ok, we'll begin by getting the start point of the second partition on the uSD card. A Raspberry Pi image has two partitions, 1 Boot, 2 Where everything else is stored. To determine the start of the second partition enter the command below;

sudo fdisk -l /media/pi/1TBFREECOM/MyImage.img

Take note of the 'Start' of the second partition '.img2', we'll need it later. Here : 131072 (see pic 1 above, red circle).

Now we mount this partition using the following command (see pic 2 above);

sudo losetup /dev/loop0 /media/pi/1TBFREECOM/MyImage.img -o $((START*512))

Where START is the start of MyImage.img2 the second partition. ie. 131072. So in this case it will be;

sudo losetup /dev/loop0 /media/pi/1TBFREECOM/MyImage.img -o $((131072*512))

Note : Don't worry if you get a busy message, just enter the following and retry the above;

sudo losetup -d /dev/loop0

Now we resize the partition with 'GParted'. From a terminal window enter the following command (see pic 3 above);

sudo gparted /dev/loop0

This will launch the 'GParted' application (see pic 4 above).

From GParted do the following;

Select /dev/loop0 partition (pic 5 above),
Select Partition menu -> Resize/Move (pic 6 above),
Resize the partition by either dragging the slider left as indicated, or entering a value into the box marked 'New size (MiB)' (again as shown in pic 6). I found this to be a little trial and error and generally started with +500M and moved up in increments. ie Change the value of "New size (MiB)" to approx 500MB above "Minimum Size". In this case 2942 + 500MB = 3442MB etc.,
Click 'Resize/Move' to finish,
GParted will now indicate there is '1 operation pending' (as in pic 7 above),
Click Edit menu -> Apply All Operations,
Click 'Apply' to warning box (pic 8 above),
GParted will now process the request to resize (see pic 9),
If the resize is successful you will get a message: 'All operations successfully completed' (as in pic 10),
Now expand the 'Applying pending operations' dialogue shown in pic 10 until you locate the line in pic 11, marked 'resize2fs -p /dev/loop0 ...' and take a note of the value. Here : '3506176K'. This is important, so, take your time to get it right.

Note : If you get an error message at step 9 (as in pic 12), then click Ok, Close, and repeat from step 1 above, only this time add 500MB to your last resize. Repeat this until the size is accepted.

It is now safe to exit GParted. Press Ctrl+Q (^Q), or menu GParted -> Quit.

We will now remove the loopback device for the second partition and create a new loopback device for the whole image. Enter the following commands (see pic 13);

sudo losetup -d /dev/loop0

sudo losetup /dev/loop0 /media/pi/1TBFREECOM/MyImage.img

Next we need to edit the partition table to reflect the new smaller size. Begin by entering the following command;

sudo fdisk /dev/loop0

For the record 'fdisk' can be tricky to use, but in a 'nutshell' we are going to do the following (pressing the return/enter key after each character entry);

Enter 'd' then '2' to delete the table entry for the second partition,
Enter 'n' then 'p' then '2' to create a new second partition entry,
Enter the START sector number that you used earlier, as the start sector. In my example it was '131072',
Enter +NEWSIZE as the new size. Don't forget the '+' sign at the start. This is the new size that you noted down before exiting GParted. Here '3506176K',
Enter 'w' to write the new partition table and exit.

Don't worry about 'Re-reading the partition table ...' error message.

Picture 14 above details what needs to be entered (the important parts are ringed in red).

Once the partition table has been written we will read the partition table to determine the end point of the second partition by issuing the following command (see pic 15 above, the end point is ringed in red);

sudo fdisk -l /dev/loop0

Pay attention to end point of the second partition.

Here it is labeled as '/dev/loop0p2' and has a value of '7143423'

Next we remove the loopback device with the following command (see pic 15 above);

sudo losetup -d /dev/loop0

Finally, we trim any empty space with the truncate command the template for which is shown below, where END is the value obtained above. Namely : 7143423

truncate -s $(((END+1)*512)) /media/pi/1TBFREECOM/MyImage.img

Consequently to execute the full command enter the following (see pic 16 above);

truncate -s $(((7143423+1)*512)) /media/pi/1TBFREECOM/MyImage.img

And that's it. Your HDD will now contain an image file named 'MyImage.img' which has been successfully resized and can by written back onto a uSD card to restore your original image should the need arise.

You can further reduce the image size by 'zipping' it, as you will get some more compression. Handy for disk storage.

Step 4: Writing Your Raspberry Pi Image

This is a similar to the read operation.

Insert the target USB uSD card into your reader

Insert your USB External HDD into your Raspberry Pi

Open an Terminal instance and enter the following Linux command;

df -h

Take note of the card mount point. '/dev/sd?'

I used a card which already had an image on it so it had two entries shown in pic 1 above and ringed in red, here; '/dev/sdb1' and '/dev/sdb2'. If you use a blank/empty or new card card it may only have one.

Now unmount your uSD card with the following command (pic 2 above);

sudo umount /dev/sdb1 /dev/sdb2

Finally write your Raspberry Pi image from the HDD '1TBFREECOM' to the uSD card with the following command (Pic 3 above).

sudo dcfldd of=/dev/sdb if=/media/pi/1TBFREECOM/MyImage.img

Follow this with a sync command, as in the case of the read (again pic 3 above).

sudo sync

That's it.

Step 5: Summary

So the steps in brief are as follows;

Set up your Raspberry Pi the way you want it, shut down the Pi and remove the uSD card,
Make a copy of the Raspberry Pi Image of the uSD to your backup HDD drive,
Resize the image,
Edit the partition table to suit the new size,
Trim the image to its new size,
Write your new image from backup HDD to a uSD card.